login as: nutanix Nutanix Controller VM nutanix@10.10.3.16's password: Last login: Wed Mar 11 07:38:11 2020 nutanix@NTNX-CVM:10.10.3.16:~$ ~/diagnostics/diagnostics.py cleanup Cleaning up node 10.10.3.13 ... done. Cleaning up node 10.10.3.14 ... done. Cleaning up node 10.10.3.15 ... done. Cleaning up the container and the storage pool ... done. nutanix@NTNX-CVM:10.10.3.16:~$ ~/diagnostics/diagnostics.py run Checking if an existing storage pool can be used ... Using storage pool default-storage-pool-87027471079717 for the tests. Checking if the diagnostics container/s exists ... Container NTNX-NFS-DEFAULT-0 does not exist. Creating a new container NTNX-NFS-DEFAULT-0 for the runs ... done. Preparing 1 UVM/s on host 10.10.3.13 ... Importing diagnostics image ... done. Deploying the UVM on host 10.10.3.13 ... done. Adding disks ... done. Preparing 1 UVM/s on host 10.10.3.14 ... Deploying the UVM on host 10.10.3.14 ... done. Adding disks ... done. Preparing 1 UVM/s on host 10.10.3.15 ... Deploying the UVM on host 10.10.3.15 ... done. Adding disks ... done. Waiting for 1 VM on host 10.10.3.13 to bootup ... done. 2 remaining. Waiting for 1 VM on host 10.10.3.14 to bootup ... done. 1 remaining. Waiting for 1 VM on host 10.10.3.15 to bootup ... done. 0 remaining. Collect executable list. Collect shared library list. Start up persistent ssh connection to localhost:17000. ERROR:root:Error connecting through control master Connecting to localhost:17000 with password authentication. Start up persistent ssh connection to localhost:17002. ERROR:root:Error connecting through control master Connecting to localhost:17002 with password authentication. Start up persistent ssh connection to localhost:17004. ERROR:root:Error connecting through control master Connecting to localhost:17004 with password authentication. Prepare environment on SVMs. Preserve original install in /home/nutanix/builds/stock. Preserve original install in /home/nutanix/builds/stock. Preserve original install in /home/nutanix/builds/stock. Transfer shared libraries. Transfer dynamic linker. Transfer executables. Run patchelf. Transferring the build across the cluster. Done! Start running test prepare_disks 2020-03-12_03-51-34: Running setup "Prepare disks" ... done. Average SVM CPU: 10.10.3.16: 70% 10.10.3.17: 64% 10.10.3.18: 69% SSD usage for disks: Disk id: 32, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 59% used Disk id: 33, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 59% used Disk id: 36, SSD used : 12 GB, SSD Capacity Bytes: 215 GB, 5% used Disk id: 37, SSD used : 15 GB, SSD Capacity Bytes: 215 GB, 7% used Disk id: 58, SSD used : 62 GB, SSD Capacity Bytes: 213 GB, 29% used Disk id: 59, SSD used : 70 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 60, SSD used : 70 GB, SSD Capacity Bytes: 270 GB, 25% used Disk id: 6154357, SSD used : 62 GB, SSD Capacity Bytes: 217 GB, 28% us ed SSD usage for storage pool: SSD tier usage 493 GB, SSD total capacity 1834 GB, SSD Usage 26% Duration prepare_disks : 49 secs ******************************************************************************* Start running test fio_seq_write Waiting for the hot cache to flush ............... done. 2020-03-12_03-53-47: Running test "Sequential write bandwidth" ... 856 MBps Average SVM CPU: 10.10.3.16: 63% 10.10.3.17: 55% 10.10.3.18: 61% SSD usage for disks: Disk id: 32, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 33, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 36, SSD used : 17 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 37, SSD used : 18 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 58, SSD used : 64 GB, SSD Capacity Bytes: 213 GB, 30% used Disk id: 59, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 60, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 6154357, SSD used : 64 GB, SSD Capacity Bytes: 217 GB, 29% us ed SSD usage for storage pool: SSD tier usage 493 GB, SSD total capacity 1834 GB, SSD Usage 26% Duration fio_seq_write : 49 secs ******************************************************************************* Start running test fio_seq_read Waiting for the hot cache to flush .............. done. 2020-03-12_03-55-54: Running test "Sequential read bandwidth" ... 3436 MBps Average SVM CPU: 10.10.3.16: 38% 10.10.3.17: 38% 10.10.3.18: 39% SSD usage for disks: Disk id: 32, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 33, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 36, SSD used : 17 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 37, SSD used : 18 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 58, SSD used : 64 GB, SSD Capacity Bytes: 213 GB, 30% used Disk id: 59, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 60, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 6154357, SSD used : 64 GB, SSD Capacity Bytes: 217 GB, 29% us ed SSD usage for storage pool: SSD tier usage 493 GB, SSD total capacity 1834 GB, SSD Usage 26% Duration fio_seq_read : 16 secs ******************************************************************************* Start running test fio_rand_read Waiting for the hot cache to flush ............... done. 2020-03-12_03-57-33: Running test "Random read IOPS" ... 244657 IOPS Average SVM CPU: 10.10.3.16: 88% 10.10.3.17: 88% 10.10.3.18: 85% SSD usage for disks: Disk id: 32, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 33, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 36, SSD used : 17 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 37, SSD used : 18 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 58, SSD used : 64 GB, SSD Capacity Bytes: 213 GB, 30% used Disk id: 59, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 60, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 6154357, SSD used : 64 GB, SSD Capacity Bytes: 217 GB, 29% us ed SSD usage for storage pool: SSD tier usage 493 GB, SSD total capacity 1834 GB, SSD Usage 26% Duration fio_rand_read : 103 secs ******************************************************************************* Start running test fio_rand_write Waiting for the hot cache to flush .............. done. 2020-03-12_04-00-35: Running test "Random write IOPS" ... 149228 IOPS Average SVM CPU: 10.10.3.16: 82% 10.10.3.17: 84% 10.10.3.18: 85% SSD usage for disks: Disk id: 32, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 33, SSD used : 129 GB, SSD Capacity Bytes: 215 GB, 60% used Disk id: 36, SSD used : 17 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 37, SSD used : 18 GB, SSD Capacity Bytes: 215 GB, 8% used Disk id: 58, SSD used : 64 GB, SSD Capacity Bytes: 213 GB, 30% used Disk id: 59, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 60, SSD used : 72 GB, SSD Capacity Bytes: 270 GB, 26% used Disk id: 6154357, SSD used : 64 GB, SSD Capacity Bytes: 217 GB, 29% us ed SSD usage for storage pool: SSD tier usage 493 GB, SSD total capacity 1834 GB, SSD Usage 26% Duration fio_rand_write : 103 secs ******************************************************************************* Tests done. Results archived in /home/nutanix/diagnostics/results/2020-03-12_03-49-01 nutanix@NTNX-CVM:10.10.3.16:~$ ^C nutanix@NTNX-CVM:10.10.3.16:~$ ~/diagnostics/diagnostics.py cleanup Cleaning up node 10.10.3.13 ... done. Cleaning up node 10.10.3.14 ... done. Cleaning up node 10.10.3.15 ... done. Cleaning up the container and the storage pool ... done.